Skip to content

[UUID 6/8] UUID multi-stage engine (planner + runtime)#18874

Open
xiangfu0 wants to merge 5 commits into
apache:masterfrom
xiangfu0:uuid-split/06-mse-planner-runtime
Open

[UUID 6/8] UUID multi-stage engine (planner + runtime)#18874
xiangfu0 wants to merge 5 commits into
apache:masterfrom
xiangfu0:uuid-split/06-mse-planner-runtime

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

What

Multi-stage (v2) engine support for UUID.

Changes

  • expressions.proto UUID / UUID_ARRAY enum values + planner type resolution, Rex parsing, proto serde, TypeFactory
  • HashJoinOperator / EnrichedHashJoinOperator + UuidLookupTable (LookupTable.normalizeKey), OneUuidKeyGroupIdGenerator, ServerPlanRequestUtils

Enables

UUID equi-joins and group-by in the multi-stage engine.

⚠️ Rolling-upgrade note: the new proto enum values are unknown to older brokers/servers; multi-stage queries with UUID literals fail on mixed-version clusters. Atomic-upgrade feature.

Depends on

#18869 and #18871.

About this PR / how to review

This is part 6 of 8 splitting #18140 (first-class logical UUID type) into layered PRs, as requested there.

The split is enabled by the v1 design: DataType.UUID has stored type BYTES, so most paths handle it automatically; each PR adds explicit UUID semantics to one subsystem. Head branch lives on xiangfu0/pinot.

This PR is stacked on #18873 (branch uuid-split/05-agg-groupby-distinct). Because GitHub PRs to apache must base on master, the Files-changed tab is cumulative (it includes layers 1–6) until the PRs below it merge. Review the commit titled [UUID 6/8] UUID multi-stage engine (planner + runtime) — that is this layer's change. Each parent merge shrinks this diff after a rebase.

Full stack (merge bottom → top)

  1. [UUID 1/8] Add logical UUID type foundation (pinot-spi) #18869 — [UUID 1/8] logical UUID type foundation (pinot-spi)
  2. [UUID 2/8] UUID ingest and segment storage #18870 — [UUID 2/8] UUID ingest and segment storage
  3. [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders) #18871 — [UUID 3/8] UUID result rendering (DataSchema, Arrow/JSON encoders)
  4. [UUID 4/8] UUID server-side predicates, CAST and transforms #18872 — [UUID 4/8] UUID server-side predicates, CAST and transforms
  5. [UUID 5/8] UUID aggregation, group-by and distinct #18873 — [UUID 5/8] UUID aggregation, group-by and distinct
  6. [UUID 6/8] UUID multi-stage engine (planner + runtime) #18874 — [UUID 6/8] UUID multi-stage engine (planner + runtime)
  7. [UUID 7/8] UUID scalar UDFs and partitioning #18875 — [UUID 7/8] UUID scalar UDFs and partitioning
  8. [UUID 8/8] UUID integration tests, benchmarks and docs #18876 — [UUID 8/8] UUID integration tests, benchmarks and docs

After #18869 lands, PRs 2/4/7 (#18870, #18872, #18875) only depend on it and can be reviewed in parallel; PRs 5/6 (#18873, #18874) also need #18871; PR 8 (#18876) needs #18874.

Full feature description, v1 design contract, scope exclusions, and benchmark numbers: #18140.

@codecov-commenter

codecov-commenter commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 18.94737% with 924 lines in your changes missing coverage. Please review.
✅ Project coverage is 37.48%. Comparing base (2892236) to head (1cf12f0).

Files with missing lines Patch % Lines
...ot/common/request/context/RequestContextUtils.java 21.21% 102 Missing and 2 partials ⚠️
...java/org/apache/pinot/common/utils/DataSchema.java 2.02% 97 Missing ⚠️
...ator/transform/function/CastTransformFunction.java 0.00% 63 Missing ⚠️
...pby/NoDictionarySingleColumnGroupKeyGenerator.java 0.00% 50 Missing ⚠️
...sform/function/ScalarTransformFunctionWrapper.java 0.00% 41 Missing ⚠️
...perator/query/NonScanBasedAggregationOperator.java 0.00% 30 Missing ⚠️
...upby/NoDictionaryMultiColumnGroupKeyGenerator.java 0.00% 30 Missing ⚠️
.../function/DistinctCountULLAggregationFunction.java 0.00% 26 Missing ⚠️
...ator/transform/function/BaseTransformFunction.java 0.00% 25 Missing ⚠️
...ransform/function/IdentifierTransformFunction.java 0.00% 25 Missing ⚠️
... and 47 more

❗ There is a different number of reports uploaded between BASE (2892236) and HEAD (1cf12f0). Click for more details.

HEAD has 16 uploads less than BASE
Flag BASE (2892236) HEAD (1cf12f0)
unittests1 1 0
unittests 2 1
java-25 5 1
temurin 5 1
integration 3 0
integration2 1 0
integration1 1 0
custom-integration1 1 0
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #18874       +/-   ##
=============================================
- Coverage     65.39%   37.48%   -27.91%     
+ Complexity     1405     1398        -7     
=============================================
  Files          3423     3426        +3     
  Lines        215968   216936      +968     
  Branches      34186    34401      +215     
=============================================
- Hits         141228    81329    -59899     
- Misses        63380   128174    +64794     
+ Partials      11360     7433     -3927     
Flag Coverage Δ
custom-integration1 ?
integration ?
integration1 ?
integration2 ?
java-25 37.48% <18.94%> (-27.91%) ⬇️
temurin 37.48% <18.94%> (-27.91%) ⬇️
unittests 37.48% <18.94%> (-27.91%) ⬇️
unittests1 ?
unittests2 37.48% <18.94%> (-0.09%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the uuid-split/06-mse-planner-runtime branch 7 times, most recently from 763c96a to 96284ae Compare July 7, 2026 07:08
@xiangfu0
xiangfu0 force-pushed the uuid-split/06-mse-planner-runtime branch 11 times, most recently from 3e1c95c to f41f8ea Compare July 14, 2026 08:03
@xiangfu0
xiangfu0 force-pushed the uuid-split/06-mse-planner-runtime branch 8 times, most recently from e827220 to 4cf327f Compare July 19, 2026 08:08
xiangfu0 added 5 commits July 20, 2026 01:08
Part 2/8 of splitting apache#18140 (logical UUID type). Rebased onto master (now includes the merged apache#18869 type foundation); stacked on 92a0d98.

Downstream references use the UuidKey class merged in apache#18869.
Part 3/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/02-ingest-storage.

Downstream references use the UuidKey class merged in apache#18869.
Part 4/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/03-result-rendering.

Downstream references use the UuidKey class merged in apache#18869.
Part 5/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/04-sse-predicates-cast.

Downstream references use the UuidKey class merged in apache#18869.
Part 6/8 of splitting apache#18140 (logical UUID type). Rebased onto latest master; stacked on uuid-split/05-agg-groupby-distinct.

Downstream references use the UuidKey class merged in apache#18869.
@xiangfu0
xiangfu0 force-pushed the uuid-split/06-mse-planner-runtime branch from 4cf327f to 1cf12f0 Compare July 20, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants